Apparatus for generating motion control signal from image signal

ABSTRACT

In order to easily generate a control signal when a motion in accordance with an image is applied to a chair for an observer, a motion vector is detected from an image signal input to an image processing apparatus by a feature information detection section. By using the motion vector, a horizontal component, a vertical component, a magnification component, and a rotation component are determined. A feature information processing section generates a control signal to be supplied to a driving section from these determined components.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image processing.

2. Description of the Related Art

An apparatus in which, when an observer is viewing an image,verisimilitude is improved by controlling, in accordance with the image,the motion of a chair on which the observer is sitting, is known. Amotion control signal, which is supplied to such an apparatus to cause achair to move, is generated from data obtained by a sensor for detectingan angle at the same time when an image is captured, or a motion controlsignal is generated through a manual operation by observing the capturedimage by a person and by predicting motion by that person.

In the above-described apparatus, when a motion control signal for achair is generated by a sensor, data for generating the motion controlsignal must be recorded at the same time the image is captured.Therefore, it is not possible to generate a motion control signal byusing an image which has already been captured. Also, when a personobserves a captured image and a motion control signal is generated, amanual operation is required, presenting a problem in that processingtakes a long time.

SUMMARY OF THE INVENTION

The present invention has been achieved in view of such circumstances.An object of the present invention is to provide an informationprocessing apparatus which generates a motion control signal from aninput image.

To achieve the above-mentioned object, according to a first aspect ofthe present invention, there is provided an information processingapparatus comprising: a motion detector for detecting a motion-relatedsignal, which is information relating to motion, in accordance with animage signal; and a generator for generating a motion control signal inaccordance with the motion-related signal.

According to a second aspect of the present invention, there is providedan information processing method comprising the steps of: detecting amotion-related signal, which is information relating to motion, inaccordance with an image signal; and generating a motion control signalin accordance with the motion-related signal.

According to a third aspect of the present invention, there is provideda storage medium storing a computer-controllable program, the programcomprising the steps of: detecting a motion-related signal, which isinformation relating to motion, in accordance with an image signal; andgenerating a motion control signal in accordance with the motion-relatedsignal.

The above and further objects, aspects and novel features of theinvention will become more apparent from the following detaileddescription when read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the construction of an embodiment ofan image processing apparatus of the present invention.

FIG. 2 is a block diagram showing the construction of a featureinformation detection section 11 of FIG. 1.

FIG. 3 shows a pattern stored in a memory 24 of FIG. 2.

FIG. 4 shows an image to be processed.

FIG. 5 is an illustration of a vector to be computed.

FIG. 6 is a block diagram showing the construction of a featureinformation processing section 12 of FIG. 1.

FIG. 7 is a side view of a driving section 3.

FIG. 8 is a view when the driving section 3 is seen from above.

DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 is a block diagram showing the construction of an embodiment ofan image processing apparatus of the present invention. An image signalsupplied from an image tape recorder (not shown), etc., is supplied to adisplay section 1 and an image processing apparatus 2. The imageprocessing apparatus 2 generates a motion control signal for driving adriving section 3.

The image processing apparatus 2 comprises a feature informationdetection section 11 and a feature information processing section 12. Asignal input to the image processing apparatus 2 is input to the featureinformation detection section 11 whereby feature information (to bedescribed later) is detected and is output to the feature informationprocessing section 12. The feature information processing section 12computes a motion control signal to be supplied to the driving section 3from the input feature information. The driving section 3 drives (motioncontrols) a chair on which an observer who observes an image sits inaccordance with the input motion control signal. This point will bedescribed later.

FIG. 2 is a block diagram showing the construction of the featureinformation detection section 11. The image signal input to the featureinformation detection section 11 is delayed by an amount correspondingto one frame by a delay section 21, after which it is supplied to aframe buffer 22-1 and is also supplied to a frame buffer 22-2. Readingsections 23-1 and 23-2 read an image signal from the corresponding framebuffers 22-1 and 22-2 in accordance with a predetermined pattern storedin a memory 24, and output the image signal to a motion vector detectionsection 25.

The motion vector detection section 25 detects a motion vector from thesupplied image signal and outputs the motion vector to a featureinformation computation section 26. The feature information computationsection 26 computes feature information from the input motion vector.

Next, the operation of the feature information detection section 11shown in FIG. 2 is described. At time t, the image signal input to thefeature information detection section 11 is supplied to the delaysection 21 and the frame buffer 22-2. The frame buffer 22-2 stores inputimage signals for one frame. Since the delay section 21 delays the imagesignal by an amount corresponding to one frame, at time t, an imagesignal at time t-1 earlier than time t, that is, an image signal oneframe before that at time t, is stored in the frame buffer 22-1. Theimage signal at time t-1, stored in the frame 22-1, is read by thereading section 23-1, and the image signal at time t, stored in theframe buffer 22-2, is read by the reading section 23-2.

The reading sections 23-1 and 23-2 read an image signal in a portioncorresponding to the pattern stored in the memory 24 from among theimage signals stored in the corresponding frame buffers 22-1 and 22-2.Here, a pattern stored in the memory 24 is described with reference toFIG. 3.

FIG. 3 shows an example of a pattern stored in the memory 24. In thecase of those portions of the pixels forming one frame, which are notrelated to motion, for example, image signals captured by a video cameramounted to an automobile, such as that shown in FIG. 4, the hood portionof the automobile is considered to be an area which is not related tomotion control, and accordingly, the pixel positioned at the center ofthe area from which that area has been removed is denoted as aconvergence point P. For example, 25 representative points Q (includingthe convergence point) at positions symmetrical horizontally andvertically about the convergence point P are set. For eachrepresentative point Q, a reference block B, composed of a predeterminednumber of pixels, with the representative point Q being positioned atthe center, for example, a reference block B composed of 33×33 pixels,is set. In the memory 24, the coordinates of each representative point Qwithin the image plane of such a frame, the size of the reference blockB, and the size of a retrieval block (not shown) composed of, forexample, 65×65 pixels, are stored as a pattern.

The reading section 23-1 reads pixel data corresponding to theabove-mentioned pattern stored in the memory 24, that is, pixel datawithin each reference block B, in accordance with the coordinates of therepresentative point Q and the size of the reference block B from amongthe image signals at time t-1 stored in the frame buffer 22-1, andoutputs the pixel data, as the data of the reference block, to themotion vector detection section 25. In a similar manner, the readingsection 23-2 reads pixel data corresponding to the pattern stored in thememory 24 from among the image signals at time t stored in the framebuffer 22-2, and outputs the pixel data, as the data of the retrievalblock, to the motion vector detection section 25.

The motion vector detection section 25 detects the motion vector at eachrepresentative point Q by performing block matching by using the inputdata of the reference block and the input data of the retrieval block.Therefore, in the case of this example, 25 motion vectors are detected.

In this embodiment, there is no need to detect a motion vector withrespect to all the pixels because the aim is to generate a motioncontrol signal. Therefore, only 25 motion vectors are determined. Thismakes it possible to reduce the scale of the circuit and to increase theprocessing speed.

The feature information computation section 26 computes, based on theequations shown below, a total of four components of a horizontalcomponent u, a vertical component v, a magnification component v_(zoom),and a rotation component v_(rot) of the motion as the entirety of theframe at time t by using the 25 motion vectors detected by the motionvector detection section 25.Horizontal component u=(1/n)Σu _(i)   (1)Vertical component v=(1/n)Σv _(i)   (2)Magnification component v _(zoom)=(1/n)Σv _(zoomi) /d _(i)   (3)Rotation component v _(rot)=(1/n)Σv _(roti) /d _(i)   (4)where the subscript i indicates the number affixed to the representativepoint Q_(i), which varies from 1 to 25 in this example, and n indicatesthe number of representative points, which is 25 in this example. Thevalues obtained by equations (1) to (4) are the average value of eachcomponent u, v, v_(zoom), and v_(rot) obtained by the motion vectordetection section 25.

The relationship among the above-mentioned components u, v, v_(zoom),and v_(rot) is described with reference to FIG. 5. The horizontalcomponent of a motion vector T of a representative point Q_(i) for theobject of processing is denoted as u_(i), and the vertical component isdenoted as v_(i). d_(i) is a scalar quantity indicating the distancefrom the convergence point P to the representative point Q_(i). (Px, Py)indicates the coordinates of the convergence point P. and the distanceup to the representative point Q_(i) of the coordinates (Q_(i)x, Q_(i)y)is computed with these coordinate values as a reference.

The component (u_(i), v_(i)) of this motion vector T is a component whenthe representative point Q_(i) is assumed to be the origin. Thecomponent in a direction parallel to the straight line which connectsthe convergence point P to the representative point Q_(i), of the motionvector T is denoted as v_(zoomi), and the component in a directionperpendicular to the straight line which connects the convergence pointP to the representative point Q_(i) is denoted as v_(roti). Also, theangle formed between the straight line which connects the convergencepoint P to the representative points Q_(i) and the motion vector T isdenoted as θ. At this time, v_(zoomi) and v_(roti) are determined basedon the following equations.v _(zoomi)=(u _(i) ² +v _(i) ²)^((1/2)) cosθ  (5)v _(roti)=(u _(i) ² +v _(i) ²)^((1/2)) sinθ  (6)

Here, although the values of 25 motion vectors are used in an averagedmanner in order to determine each component, each component may beweighted based on the positional relationship on the image plane.

The feature information computation section 26 computes four componentdata u, v, v_(zoom), and v_(rot), as feature information, by usingequations (1) to (4) from the motion vector output from the motionvector detection section 25. The computed four component data u, v,v_(zoom), and v_(rot) are output to the feature information processingsection 12 (FIG. 1).

Here, for example, when an observer is made to virtually experience thefeeling of riding in an automobile, the types of forces (motions) whichshould be applied to the chair on which the observer is sitting areconsidered. Examples of forces applied to the chair of the automobileinclude a force for representing the forward-rearward inclination of thesurface of the road when the automobile is running on a sloping road,such as, a grade, a force for representing vibrations in the up-and-downdirection which are received from the surface of the road when theautomobile is running on a bumpy road, and a force for representing theinclination in the left-right direction of the surface of the road whenthe automobile is running on a sloping road.

These forces are forces capable of providing a stimulus in a form inwhich the physical sensation is the same, to the chair of the observerwho observes the image, from among the stimuli given to the automobileto which a video camera capturing the image is mounted, and are called“actual stimuli” herein. The actual stimulus is such that when the valuethereof is integrated, the value becomes zero.

In contrast, a force for representing the centrifugal force when theautomobile is turning on a curve, a force for representing the inertialforce during acceleration and deceleration, and a force for representingthe yawing of the automobile in a curve are such that the value thereofdoes not become zero even if the value is integrated. These are stimuliin which it is difficult to give in a form in which the physicalsensation is the same as the stimuli which are given to the automobiledue to limitations in the distance the chair can be moved, limitationsin the movement direction, etc., and are called “simulated stimuli”herein.

The relationship among the force concerning the above-mentioned actualstimulus and the simulated stimulus, the component of the motion controlsignal which is actually applied to the chair for the observer, and thefour components computed by the feature information computation section26 is shown below. The motion control signal for the forward-rearwardinclination of the surface of the road among the actual stimuli shownbelow is one of the motion control signal components “pitch” and can berepresented by an amount of the low frequencies from among thecomponents in the vertical direction of the motion vector. Here, sincethe inclination of the surface of the road is considered to vary in slowcycles, low-frequency components are used. The vibration received fromthe surface of the road is considered to be given in a verticaldirection and is also considered to be at a high frequency. Therefore,for the motion control signal for the vibration received from thesurface of the road, high-frequency components among the components inthe vertical direction of the motion vector are used. This value isrepresented as a motion control signal component z.

The motion control signal for the left-right inclination of the surfaceof the road is one of the motion control signal components “roll”. As isclear from FIG. 5, the rotation component v_(roti) can be represented bya value such that 25 motion vectors are added together. The motioncontrol signal for the centrifugal force in a curve is one of the motioncontrol signal components “roll” and is represented by a horizontalcomponent u. The motion control signal for the inertial force byacceleration and deceleration is one of the motion control signalcomponents “pitch” and is represented by low-frequencies of thedifferentiated value of the magnification component v_(zoom). The reasonthe motion control signal is a low-frequency component is that asensitive operation is unnecessary for acceleration and deceleration.The motion control signal for the yawing of the automobile in a curve isone of the motion control signal components “yaw” and is represented bya horizontal component u. The reason the motion control signal is −u isthat the motion control signal acts in reverse to the centrifugal forcein a curve.

Actual Stimulus Component to Motion control Relationship be representedsignal component with 4 components Actual stimulus Forward-rearwardpitch Low-frequency component inclination of of Σv surface of roadVibrations received z High-frequency component from surface of road of−Σv Left-right roll −Σv_(rot) inclination of surface of road Simulatedstimulus Centrifugal force roll u in curve Inertial force due pitchLow-frequency components to acceleration and of dv_(zoom)/dtdeceleration Automobile yaw −u yawing in curve

The feature information processing section 12 generates a motion controlsignal to be supplied to the driving section 3 (FIG. 1) by using theabove-described relationships. FIG. 6 is a block diagram showing theconstruction of the feature information processing section 12. Among thefour components output from the feature information detection section11, the rotation component v_(rot) is input to an adder 31-1, thehorizontal component u is input to an adder 31-2 and a code inverter32-1, the vertical component v is input to an adder 31-3, and themagnification component v_(zoom) is input to an adder 31-5 and a delayunit 33-3. The data output from the adder 31-1 is delayed by an amountcorresponding to one clock by the delay unit 33-1, after which it is fedback and is input to the adder 31-1. In a similar manner, the data whichis output from the adder 31-3 is delayed by an amount corresponding toone clock by a delay unit 33-2, after which it is fed back and is inputto the adder 31-3 as well.

The data output from the delay unit 33-1 is input to the adder 31-2, andthe data output from the delay unit 33-2 is output to an HPF (High-PassFilter) 34 via a code inverter 32-2 and is also output to an adder 31-4via an LPF (Low-Pass Filter) 35-1. The magnification component v_(zoom)which is delayed by an amount corresponding to one clock by the delayunit 33-3 is subtracted from the magnification component v_(zoom) inputto an adder 31-5, and the resulting component is input to an adder 31-4via an LPF 35-2.

Next, a description is given of the computation of the motion controlsignal components “roll”, “yaw”, “z”, and “pitch”, performed by thefeature information processing section 12. The rotation componentv_(rot) input to the feature information processing section 12 is inputto the adder 31-1. The adder 31-1 adds together the rotation componentv_(rot) which is input at time t and the data which is output one framebefore at time t-1 from the delay unit 33-1. The adder 31-1 computes themotion control signal component “roll” (Σv_(rot)) which represents theleft-right inclination of the surface of the road by accumulating(integrating) the rotation component v_(rot) in this manner. However,since the motion control signal component “roll” representing theleft-right inclination of the surface of the road is −Σv_(rot), theadder 31-2 uses, for computation, data such that the code of the datainput from the delay unit 33-1 is inverted.

The motion control signal component “roll” (horizontal component u) isalso used to represent the centrifugal force in a curve. Accordingly,the adder 31-2 computes the motion control signal component “roll” to besupplied to the driving section 3 by adding together (subtracting theoutput of the delay unit 33-1 from the horizontal component u) the datasuch that the code of the data input from the delay unit 33-1 isinverted and the horizontal component u.

Since the motion control signal component “yaw” of the yawing of theautomobile in a curve is obtained by inverting the value of thehorizontal component u, the feature information processing section 12computes the motion control signal component “yaw” by causing the codeinverter 32-1 to invert the code of the value of the input horizontalcomponent u.

The adder 31-3 adds together the vertical component v which is input attime t and the vertical component v which is output one frame before attime t-1 from the delay unit 33-2. In this manner, the verticalcomponent v is accumulated (integrated) by the adder 31-3. Then, thedata which is accumulated by the adder 31-3 and the delay unit 33-2 isinput to the code inverter 32-2 whereby the code is inverted, andfurther, only the high-frequency components are extracted by the HPF 34.In this manner, the motion control signal component z representing thevibrations received from the surface of the road is computed.

Furthermore, the data output from the delay unit 33-2 is also output tothe LPF 35-1 whereby the low-frequency components are extracted. In thismanner, the motion control signal component “pitch” representing theforward-rearward inclination of the surface of the road is computed. Themotion control signal component “pitch” is also used as a motion controlsignal component representing the inertial force by acceleration anddeceleration. For this reason, the motion control signal component“pitch” output from the LPF 35-1 is added to the motion control signalcomponent “pitch” representing the inertial force by the adder 31-4.

The motion control signal component “pitch” representing the inertialforce is computed from the magnification component v_(zoom) input to thefeature information processing section 12. The magnification componentv_(zoom) input to the feature information processing section 12 is inputto the adder 31-5 and the delay unit 33-3. A magnification componentv_(zoom)t which is input at time t and a magnification componentv_(zoom)t-1 which is input at time t-1, delayed by one frame by thedelay unit 33-3, are input to the adder 31-5. The adder 31-5differentiates the magnification component v_(zoom) by subtracting themagnification component v_(zoom)t-1 at time t-1 from the magnificationcomponent v_(zoom)t input at time t. Then, the LPF 35-2 extracts thelow-frequency components from the value output from the adder 31-5,thereby computing the motion control signal component “pitch”representing the inertial force by acceleration and deceleration.

The adder 31-4 adds together the value output from the LPF 35-1 and thevalue output from the LPF 35-2, thereby computing the motion controlsignal component “pitch” to be supplied to the driving section 3.

An example of the driving section 3 is shown in FIGS. 7 and 8. FIG. 7 isa side view of the driving section 3. FIG. 8 is a view when the drivingsection 3 is seen from above. The driving section 3 comprises sixpistons 41-1 to 41-6 serving as actuators, with a base 42 beingsupported by these pistons. The base 42 has a chair 43 fixed thereto sothat an observer 44 may sit on this chair 43.

The pistons 41-1 to 41-6 are capable of extending and retracting alongtheir central axes. As a result of the extending and retracting motionby the pistons 41-1 to 41-6, the base 42 jerks, and furthermore, thechair 43 fixed to the base 42 jerks. A signal for controlling thepistons 41-1 to 41-6 is generated and supplied by the featureinformation processing section 12 in a manner as described above.

Table 1 shows a specific example of the operation by the driving section3. The upward arrow in this Table 1 indicates that the piston extends,and the downward arrow indicates that the piston retracts. TABLE 1Actual stimulus Motion control Component to be signal representedcomponent 41-1 41-2 41-3 41-6 41-5 41-4 Forward- Pitch Fwd ↓ Fwd ↓ Fwd ↑Fwd ↓ Fwd ↓ Fwd ↑ rearward Rwd ↑ Rwd ↑ Rwd ↓ Rwd ↑ Rwd ↑ Rwd ↓inclination of surface of road Vibration z ↑ ↓ ↑ ↓ ↑ ↓ ↑ ↓ ↑ ↓ ↑ ↓received from surface of road Left-right roll Left ↓ Left ↓ Left ↓ Left↑ Left ↑ Left ↑ inclination of Right↑ Right↑ Right↑ Right↓ Right↓ Right↓surface of roadFwd (Forward): down slopeRwd (Rearward): up slope↑ ↓: periodically repeatsRight: portion on right side along travelling direction is lowLeft: portion on left side along travelling direction is low

Simulated stimulus Component to be represented A B C D E F Centrifugalroll Left ↑ Left ↑ Left ↑ Left ↓ Left ↓ Left ↓ force in curve Right↓Right↓ Right↓ Right↑ Right↑ Right↑ Inertial force pitch Dclt ↓ Dclt ↓Dclt ↑ Dclt ↓ Dclt ↓ Dclt ↑ due to acceleration and Aclt ↑ Aclt ↑ Aclt ↓Aclt ↑ Aclt ↑ Aclt ↓ deceleration Yawing of yaw Left ↑ Left ↓ Left ↑Left ↓ Left ↑ Left ↓ automobile in Right↓ Right↑ Right↓ Right↑ Right↓Right↑ curveLeft: automobile turns leftRight: automobile turns rightDclt: deceleratesAclt: accelerates

As described above, in this embodiment, the motion control signalcomponent is computed from the motion obtained from the image. Thismakes it possible to obviate the need to determine the motion controlsignal component in order to input it in advance by observing the imageby a person. In addition, use of a sensor, etc., makes it possible toeasily generate a motion control signal component from the image inwhich a motion control signal component is not generated.

Although in this embodiment a motion vector is detected, it is alsopossible to detect only the direction of the motion or only the quantityof the motion, depending on the application.

Although in the above-described example an example by hardware is shown,the operation of the present invention may be provided as a programwhich is computer-controllable software. This program may be supplied toan information processing apparatus which is controlled by a computer byusing an information storage medium, such as a magnetic disk, an opticaldisk such as a CD-ROM, magnetic tape, etc., or by using a storagemedium, such as a transmission medium utilizing a network using theInternet, a digital satellite, etc.

This program may comprise a feature information detecting step, afeature information processing step, and a driving step which performprocessing which is the same as the processing performed by the featureinformation detection section 11, the feature information processingsection 12, and the driving section 3, which is a realization example byhardware shown in FIG. 1, and these steps may be performed, for example,for each frame.

Many different embodiments of the present invention may be constructedwithout departing from the spirit and scope of the present invention. Itshould be understood that the present invention is not limited to thespecific embodiment described in this specification. To the contrary,the present invention is intended to cover various modifications andequivalent arrangements included within the spirit and scope of theinvention as hereafter claimed. The scope of the following claims is tobe accorded the broadest interpretation so as to encompass all suchmodifications, equivalent structures and functions.

1-23. (canceled)
 24. An information processing apparatus comprising: amotion detector for detecting motion vectors for a plurality ofpredetermined blocks within each frame of an image signal to bedisplayed by a display device; a generator for generating a motioncontrol signal corresponding to each frame of said image signal bycalculating said motion vectors; the motion control signal beinggenerated to represent both actual stimulus and simulated stimulus to anobject; the simulated stimulus including at least componentscorresponding to centrifugal force, inertial force, and yaw; whereinsaid generator generates, as said motion control signal, a horizontalcomponent, a vertical component, a magnification component, and arotation component in accordance with said motion vectors; and a drivingdevice for driving the object in accordance with said motion controlsignal, whereby the movement of the driven object is controlled by themotion control signal in a manner imparting both actual and simulatedstimulus to the object.
 25. An information processing apparatuscomprising: a motion detector for detecting motion vectors for aplurality of predetermined blocks within each frame of an image signalto be displayed by a display device; a generator for generating a motioncontrol signal corresponding to each frame of said image signal bycalculating said motion vectors; the motion control signal beinggenerated to represent both actual stimulus and simulated stimulus to anobject; the simulated stimulus including at least componentscorresponding to centrifugal force, inertial force, and yaw; and adriving device for driving the object in accordance with said motioncontrol signal, whereby the movement of the driven object is controlledby the motion control signal in a manner imparting both actual andsimulated stimulus to the object; wherein a chair is provided as saidobject, and said driving device comprises an actuator for moving saidchair.
 26. An information processing method comprising the steps of:detecting motion vectors for a plurality of predetermined blocks withineach frame of an image signal to be displayed by a display device;generating a motion control signal corresponding to each frame of saidimage signal by calculating said motion vectors; the motion controlsignal being generated to represent both actual stimulus and simulatedstimulus to an object; the simulated stimulus including at leastcomponents corresponding to centrifugal force, inertial force, and yaw;wherein, in said generating step, as said motion control signal, ahorizontal component, a vertical component, a magnification component,and a rotation component are detected in accordance with said motionvectors; and driving the object in accordance with said motion controlsignal, whereby the movement of the driven object is controlled by themotion control signal in a manner imparting both actual and simulatedstimulus to the object.
 27. A storage medium storing acomputer-controllable program, said program comprising the steps of:detecting motion vectors for a plurality of predetermined blocks withineach frame of an image signal to be displayed by a display device;generating a motion control signal corresponding to each frame of saidimage signal by calculating said motion vectors; the motion controlsignal being generated to represent both actual stimulus and simulatedstimulus to an object; the simulated stimulus including at leastcomponents corresponding to centrifugal force, inertial force, and yaw;wherein, in said generating step, as said motion control signal, ahorizontal component, a vertical component, a magnification component,and a rotation component are detected in accordance with said motionvectors; and driving the object in accordance with said motion controlsignal, whereby the movement of the driven object is controlled by themotion control signal in a manner imparting both actual and simulatedstimulus to the object.